Problem Statement¶
Business Context¶
Workplace safety in hazardous environments like construction sites and industrial plants is crucial to prevent accidents and injuries. One of the most important safety measures is ensuring workers wear safety helmets, which protect against head injuries from falling objects and machinery. Non-compliance with helmet regulations increases the risk of serious injuries or fatalities, making effective monitoring essential, especially in large-scale operations where manual oversight is prone to errors and inefficiency.
To overcome these challenges, SafeGuard Corp plans to develop an automated image analysis system capable of detecting whether workers are wearing safety helmets. This system will improve safety enforcement, ensuring compliance and reducing the risk of head injuries. By automating helmet monitoring, SafeGuard aims to enhance efficiency, scalability, and accuracy, ultimately fostering a safer work environment while minimizing human error in safety oversight.
Objective¶
As a data scientist at SafeGuard Corp, you are tasked with developing an image classification model that classifies images into one of two categories:
- With Helmet: Workers wearing safety helmets.
- Without Helmet: Workers not wearing safety helmets.
Data Description¶
The dataset consists of 631 images, equally divided into two categories:
- With Helmet: 311 images showing workers wearing helmets.
- Without Helmet: 320 images showing workers not wearing helmets.
Dataset Characteristics:
- Variations in Conditions: Images include diverse environments such as construction sites, factories, and industrial settings, with variations in lighting, angles, and worker postures to simulate real-world conditions.
- Worker Activities: Workers are depicted in different actions such as standing, using tools, or moving, ensuring robust model learning for various scenarios.
Installing and Importing the Necessary Libraries¶
In [1]:
!pip install tensorflow[and-cuda] numpy==1.25.2 -q
Installing build dependencies ... done error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.
In [2]:
import tensorflow as tf
print("Num GPUs Available:", len(tf.config.list_physical_devices('GPU')))
print(tf.__version__)
Num GPUs Available: 1 2.19.0
Note:
After running the above cell, kindly restart the notebook kernel (for Jupyter Notebook) or runtime (for Google Colab) and run all cells sequentially from the next cell.
On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.
In [3]:
import os
import random
import numpy as np # Importing numpy for Matrix Operations
import pandas as pd
import seaborn as sns
import matplotlib.image as mpimg # Importing pandas to read CSV files
import matplotlib.pyplot as plt # Importting matplotlib for Plotting and visualizing images
import math # Importing math module to perform mathematical operations
import cv2
# Tensorflow modules
import keras
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator # Importing the ImageDataGenerator for data augmentation
from tensorflow.keras.models import Sequential # Importing the sequential module to define a sequential model
from tensorflow.keras.layers import Dense,Dropout,Flatten,Conv2D,MaxPooling2D,BatchNormalization # Defining all the layers to build our CNN Model
from tensorflow.keras.optimizers import Adam,SGD # Importing the optimizers which can be used in our model
from sklearn import preprocessing # Importing the preprocessing module to preprocess the data
from sklearn.model_selection import train_test_split # Importing train_test_split function to split the data into train and test
from sklearn.metrics import confusion_matrix
from tensorflow.keras.models import Model
from keras.applications.vgg16 import VGG16 # Importing confusion_matrix to plot the confusion matrix
# Importing callbacks to improve training:
# - EarlyStopping: stops training when validation loss stops improving to prevent overfitting
# - ReduceLROnPlateau: reduces learning rate when validation loss stagnates to help escape plateaus
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
# Import the GlobalAveragePooling2D layer used to reduce spatial dimensions from feature maps
from tensorflow.keras.layers import GlobalAveragePooling2D
# Display images using OpenCV
from google.colab.patches import cv2_imshow
#Imports functions for evaluating the performance of machine learning models
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report
from sklearn.metrics import mean_squared_error as mse # Importing cv2_imshow from google.patches to display images
from sklearn.metrics import (
confusion_matrix, classification_report, roc_curve, roc_auc_score,
precision_recall_curve, average_precision_score,
accuracy_score, precision_score, recall_score, f1_score
)
# Ignore warnings
import warnings
warnings.filterwarnings('ignore')
In [4]:
# Set the seed using keras.utils.set_random_seed. This will set:
# 1) `numpy` seed
# 2) backend random seed
# 3) `python` random seed
tf.keras.utils.set_random_seed(812)
Data Overview¶
Loading the data¶
In [5]:
# Import the drive module from Google Colab to access Google Drive
from google.colab import drive
# Mount Google Drive to the Colab environment at the specified path
# This will prompt the user to authorize access to their Google Drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [6]:
# Load a NumPy array of images from Google Drive
images = np.load('/content/drive/MyDrive/Academics Great Learning/University of Texas/Computer_Vision/images_proj.npy')
# Load a CSV file containing labels into a pandas DataFrame
labels = pd.read_csv('/content/drive/MyDrive/Academics Great Learning/University of Texas/Computer_Vision/Labels_proj.csv')
In [7]:
print(f"Dataset shape: {images.shape}")
print(f"Labels shape: {labels.shape}")
print(f"Image data type: {images.dtype}")
print(f"Image value range: [{images.min()}, {images.max()}]")
print(f"Unique labels: {np.unique(labels)}")
print(f"Number of samples: {len(images)}")
print(f"Image shape (H, W, C): {images.shape[1:]}")
print(f"Number of classes: {len(np.unique(labels))}")
print(f"Mean pixel value: {images.mean():.2f}, Std: {images.std():.2f}")
print(f"Dataset size (MB): {images.nbytes / 1024**2:.2f}")
Dataset shape: (631, 200, 200, 3) Labels shape: (631, 1) Image data type: uint8 Image value range: [0, 255] Unique labels: [0 1] Number of samples: 631 Image shape (H, W, C): (200, 200, 3) Number of classes: 2 Mean pixel value: 128.91, Std: 70.69 Dataset size (MB): 72.21
Exploratory Data Analysis¶
Plot random images from each of the classes and print their corresponding labels.¶
In [8]:
def _to_hwc(img):
"""
Ensure image is (H, W) or (H, W, C).
Accepts (H, W), (H, W, C), or (C, H, W) where C in {1,3,4}.
"""
arr = np.asarray(img)
if arr.ndim == 2:
return arr # (H, W)
if arr.ndim == 3:
# (H, W, C)
if arr.shape[-1] in (1, 3, 4):
return arr
# (C, H, W)
if arr.shape[0] in (1, 3, 4):
return np.transpose(arr, (1, 2, 0))
raise ValueError(f"Unsupported image shape: {arr.shape}. Expected (H,W), (H,W,C), or (C,H,W).")
def plot_sample_images_by_class(
images,
labels,
n_samples_per_class=8,
class_names=None,
random_state=None,
show_colorbar=False
):
"""
Plot random images from each class and print corresponding labels + metadata.
Parameters
----------
images : np.ndarray
Image array of shape (N, H, W[, C]) or (N, C, H, W). dtype can be uint8/float*.
labels : np.ndarray
Integer labels of shape (N,).
n_samples_per_class : int, default 8
Number of samples to draw per class (uses min(count, n_samples_per_class)).
class_names : dict or list, optional
Mapping {label_int: "name"} or list indexed by label. If None, uses str(label).
random_state : int, optional
Seed for reproducible sampling.
show_colorbar : bool, default False
Add a colorbar per image (can clutter large grids).
"""
rng = np.random.default_rng(random_state)
labels = np.asarray(labels)
n = len(images)
if len(labels) != n:
raise ValueError(f"images and labels length mismatch: {n} vs {len(labels)}")
# Unique classes sorted
classes = np.unique(labels)
n_classes = len(classes)
# Prepare sampling indices per class
sampled_indices_per_class = {}
for c in classes:
idx = np.where(labels == c)[0]
if len(idx) == 0:
continue
k = min(n_samples_per_class, len(idx))
sampled = rng.choice(idx, size=k, replace=False)
sampled_indices_per_class[c] = sampled
# Determine grid size (rows = classes, cols = max samples actually drawn)
max_k = max(len(v) for v in sampled_indices_per_class.values()) if sampled_indices_per_class else 0
if max_k == 0:
raise ValueError("No samples found for any class.")
fig, axes = plt.subplots(n_classes, max_k, figsize=(2.6*max_k, 2.6*n_classes))
if n_classes == 1 and max_k == 1:
axes = np.array([[axes]])
elif n_classes == 1:
axes = axes[np.newaxis, :]
elif max_k == 1:
axes = axes[:, np.newaxis]
# Collect metadata rows to print after plotting
meta_rows = []
for r, c in enumerate(classes):
row_axes = axes[r]
sampled = sampled_indices_per_class.get(c, [])
cname = (
class_names[c] if isinstance(class_names, dict) and c in class_names else
(class_names[c] if isinstance(class_names, (list, tuple)) and 0 <= c < len(class_names) else str(c))
)
for col in range(max_k):
ax = row_axes[col]
ax.axis('off')
if col >= len(sampled):
ax.set_title(f"{cname}\n(no sample)", fontsize=9)
continue
idx = sampled[col]
img = _to_hwc(images[idx])
# Pick cmap for grayscale
cmap = 'gray' if (img.ndim == 2 or (img.ndim == 3 and img.shape[-1] == 1)) else None
shown = img.squeeze() if (img.ndim == 3 and img.shape[-1] == 1) else img
im = ax.imshow(shown, cmap=cmap)
ax.set_title(f"{cname} | Index: {idx}", fontsize=10)
if show_colorbar:
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
# Compute quick stats
arr = np.asarray(img, dtype=np.float32)
vmin, vmax = float(arr.min()), float(arr.max())
vmean, vstd = float(arr.mean()), float(arr.std())
meta_rows.append({
"index": int(idx),
"class": int(c),
"class_name": cname,
"shape": tuple(img.shape),
"dtype": str(np.asarray(images[idx]).dtype),
"min": vmin,
"max": vmax,
"mean": round(vmean, 4),
"std": round(vstd, 4),
})
plt.suptitle("Random Samples by Class", fontsize=14)
plt.tight_layout(rect=[0, 0, 1, 0.97])
plt.show()
# Print a neat metadata table sorted by class then index
meta_df = pd.DataFrame(meta_rows).sort_values(["class", "index"]).reset_index(drop=True)
print("\nSelected sample metadata (per image):")
print(meta_df.to_string(index=False))
return meta_df # return the table in case you want to use it further
# ---- Example usage ----
# meta = plot_sample_images_by_class(
# images, labels,
# n_samples_per_class=8,
# class_names={0: "Without Helmet", 1: "With Helmet"},
# random_state=42,
# show_colorbar=False
# )
In [9]:
meta = plot_sample_images_by_class(
images,
labels,
n_samples_per_class=8, # how many images per class
class_names={0: "Without Helmet", 1: "With Helmet"},
random_state=42, # for reproducibility
show_colorbar=False # set True if you want color scales
)
Selected sample metadata (per image):
index class class_name shape dtype min max mean std
298 0 Without Helmet (200, 200, 3) uint8 4.0 255.0 166.9986 63.6311
408 0 Without Helmet (200, 200, 3) uint8 0.0 255.0 132.8423 73.2593
409 0 Without Helmet (200, 200, 3) uint8 0.0 224.0 98.2472 53.2255
477 0 Without Helmet (200, 200, 3) uint8 10.0 255.0 156.5704 63.6916
494 0 Without Helmet (200, 200, 3) uint8 20.0 252.0 105.8544 44.5249
514 0 Without Helmet (200, 200, 3) uint8 3.0 255.0 136.1561 64.1865
544 0 Without Helmet (200, 200, 3) uint8 0.0 233.0 73.1376 66.2615
629 0 Without Helmet (200, 200, 3) uint8 6.0 224.0 111.0633 50.0075
39 1 With Helmet (200, 200, 3) uint8 0.0 255.0 169.6184 68.7074
56 1 With Helmet (200, 200, 3) uint8 0.0 255.0 108.9386 62.5875
114 1 With Helmet (200, 200, 3) uint8 0.0 255.0 86.4737 61.4922
138 1 With Helmet (200, 200, 3) uint8 0.0 255.0 125.5381 64.6809
154 1 With Helmet (200, 200, 3) uint8 0.0 255.0 163.9550 58.0341
156 1 With Helmet (200, 200, 3) uint8 0.0 255.0 196.1156 79.0397
238 1 With Helmet (200, 200, 3) uint8 0.0 255.0 144.9041 65.8665
257 1 With Helmet (200, 200, 3) uint8 0.0 255.0 153.4297 74.4420
Observations from the Shown Samples¶
- The grid is balanced, with equal examples from both classes: Without Helmet and With Helmet.
- The images cover diverse scenarios, including construction sites and industrial settings.
- There is noticeable variation in lighting conditions, camera angles, and worker postures across the shown samples.
- Workers are depicted in multiple activities — standing, inspecting, operating machinery, or moving within the scene.
- Image format appears consistent (200×200×3,
uint8) and resolution looks uniform across the displayed items. - A few samples exhibit strong color casts/filters (e.g., bluish tones or posterized effects), indicating possible preprocessing or source artifacts.
- Framing varies: some Without Helmet images are tight face crops, while several With Helmet images show wider, context-rich views.
- Background complexity ranges from simple (plain or blurred) to cluttered (machinery, structures), which may influence model attention.
- Pixel statistics printed below the grid show substantial per-image intensity spread (mean/std), suggesting diverse lighting/contrast within these samples.
- At least one sample appears stock-like or heavily edited; if common, such images may need review to ensure real-world relevance.
Checking for class imbalance¶
In [10]:
def plot_class_distribution(labels, class_names=None):
"""
Clean bar plot of class distribution with counts + percentages.
Robust to labels being strings ('0','1') or ints (0,1).
"""
# 1) Get 1D labels
if isinstance(labels, pd.DataFrame):
if labels.shape[1] != 1:
raise ValueError("`labels` DataFrame has multiple columns; pass a single column or a Series.")
y_raw = labels.iloc[:, 0].to_numpy()
elif isinstance(labels, pd.Series):
y_raw = labels.to_numpy()
else:
y_raw = np.asarray(labels)
y_raw = y_raw.ravel()
# 2) Prefer numeric if possible (for stats), but we will PLOT AS STRINGS to avoid palette issues
y_num = pd.to_numeric(y_raw, errors="coerce")
numeric_ok = np.isfinite(y_num).all()
y_for_stats = y_num.astype(int) if numeric_ok else y_raw.astype(str)
# 3) Counts/percentages (keep class order sorted)
counts = pd.Series(y_for_stats).value_counts().sort_index()
classes = counts.index.tolist() # classes used for stats
classes_str = [str(c) for c in classes] # classes used for plotting (strings only)
N, K = int(counts.sum()), len(classes)
perc = (counts / N * 100).round(2)
# 4) Imbalance / class_weight (printed, not drawn)
maj = counts.idxmax()
min_ = counts.idxmin()
imbalance_ratio = counts.max() / counts.min() if counts.min() > 0 else np.inf
class_weight = {cls: (N / (K * int(cnt))) for cls, cnt in counts.items()}
# 5) Choose bar colors in-order to avoid dict-key errors
if set(classes_str) == {"0", "1"}:
color_map = {"0": "tomato", "1": "mediumseagreen"}
colors = [color_map[c] for c in classes_str]
xticklabels = ["Without Helmet (0)", "With Helmet (1)"]
else:
colors = None
xticklabels = classes_str
# 6) Plot (uncluttered)
plt.figure(figsize=(7, 5))
sns.set_style("whitegrid")
ax = sns.barplot(x=classes_str, y=counts.values, palette=colors)
# annotate counts + %
for i, p in enumerate(ax.patches):
h = int(p.get_height())
ax.annotate(f"{h}\n({perc.iloc[i]}%)",
(p.get_x() + p.get_width()/2., h),
ha='center', va='bottom', fontsize=11)
# balanced reference line (no text label)
ax.axhline(N / K, linestyle='--', linewidth=1, color='blue')
ax.set_xlabel("Class Labels", fontsize=12)
ax.set_ylabel("Number of Images", fontsize=12)
ax.set_title("Helmet Classification: Image Counts per Class", fontsize=14)
ax.set_xticklabels(xticklabels, fontsize=11)
# 👉 Force y-axis from 0 to 350
ax.set_ylim(0, 350)
plt.tight_layout()
plt.show()
# 7) Print extra info (kept out of figure)
print("📊 Dataset Summary")
print(f"Total samples: {N}")
print(f"Classes: {K}")
print("Class distribution:")
for i, cls in enumerate(classes):
# Resolve pretty names if provided
if class_names:
try:
key = int(cls) if isinstance(cls, (int, np.integer, np.int64)) or str(cls).isdigit() else cls
name = class_names.get(key, str(cls))
except Exception:
name = class_names.get(cls, str(cls))
else:
name = str(cls)
print(f" {name}: {counts.iloc[i]} ({perc.iloc[i]}%)")
print(f"\nImbalance Ratio (majority/minority): {imbalance_ratio:.2f}")
print("Suggested sklearn class_weight:", class_weight)
# Example
plot_class_distribution(labels, class_names={0: "Without Helmet", 1: "With Helmet"})
📊 Dataset Summary
Total samples: 631
Classes: 2
Class distribution:
Without Helmet: 320 (50.71%)
With Helmet: 311 (49.29%)
Imbalance Ratio (majority/minority): 1.03
Suggested sklearn class_weight: {0: 0.9859375, 1: 1.0144694533762058}
Observations:¶
Balanced dataset
- The two classes are very close in size: 320 vs 311 samples.
- Imbalance ratio ≈ 1.03 (negligible).
Proportional representation
- Without Helmet: 50.7%
- With Helmet: 49.3%
- Nearly even split; good for training.
No serious imbalance
- Oversampling/undersampling not needed for most models.
- Suggested
class_weight:{0: 0.9859, 1: 1.0144}(both ~1).
Dataset size
- Total images: 631 (moderate).
- Consider data augmentation to improve generalization.
Balanced reference check
- Ideal per-class count ≈ 315; both classes are close (320, 311).
Metric interpretability
- With near-balance, accuracy, precision, and recall are all meaningful (accuracy not misleading).
Conclusion: The dataset is well balanced between classes. You can train without special imbalance handling; add data augmentation (flips, rotations, brightness/contrast) to boost robustness.
Data Preprocessing¶
Converting images to grayscale¶
In [11]:
# Convert RGB images to Grayscale
def rgb_to_grayscale(images):
"""Convert RGB images to grayscale using weighted average."""
# 0.299*R + 0.587*G + 0.114*B
grayscale_images = np.dot(images[..., :3], [0.299, 0.587, 0.114])
return grayscale_images.astype(np.float32) # keep float, scale later if needed
# Convert to grayscale
images_gray = rgb_to_grayscale(images)
print(f"Grayscale images shape: {images_gray.shape}") # (N, H, W)
# Plot before and after preprocessing
fig, axes = plt.subplots(2, 6, figsize=(18, 8))
sample_indices = np.random.choice(len(images), 6, replace=False)
for i, idx in enumerate(sample_indices):
# Original RGB
axes[0, i].imshow(images[idx].astype(np.uint8))
axes[0, i].set_title(f'Original RGB\nLabel: {labels[idx] if not hasattr(labels, "iloc") else labels.iloc[idx, 0]}')
axes[0, i].axis('off')
# Grayscale
axes[1, i].imshow(images_gray[idx], cmap='gray')
axes[1, i].set_title(f'Grayscale\nLabel: {labels[idx] if not hasattr(labels, "iloc") else labels.iloc[idx, 0]}')
axes[1, i].axis('off')
plt.suptitle('Before and After Preprocessing: RGB to Grayscale', fontsize=16)
plt.tight_layout()
plt.show()
# Reshape grayscale images for CNN (add channel dimension)
images_processed = images_gray[..., np.newaxis] # safer than reshape
print(f"Processed images shape: {images_processed.shape}") # (N, H, W, 1)
Grayscale images shape: (631, 200, 200)
Processed images shape: (631, 200, 200, 1)
Observations (updated for the improved grayscale pipeline)¶
I kept the grayscale conversion NumPy-vectorized to stay consistent with our project’s emphasis on clean, efficient array ops.
Efficiency via vectorization
Using a singlenp.dot(images[..., :3], [0.299, 0.587, 0.114])converts the entire batch at once (no Python loops), which scales well to large datasets.Standards-based conversion
The weights 0.299/0.587/0.114 follow the common luminance formula, making the transformation transparent and reproducible.Numerical stability for ML
The grayscale tensor is kept asfloat32(not immediately cast touint8). This avoids truncation/clipping during preprocessing and is better for normalization (e.g.,/255.0) and model training.CNN-ready shape without brittle reshape
The channel dimension is added withimages_gray[..., np.newaxis], yielding (N, H, W, 1). This is safer/readable vs. manual reshapes and aligns with TensorFlow/Keras defaults.Plotting correctness
For visualization, the original RGB frames are explicitly cast touint8to render correctly, while grayscale frames are shown withcmap='gray'to ensure consistent display.Robust label access
Label indexing is handled to support both NumPy arrays and Pandas DataFrames (Series/single-columnDataFrame), preventing indexing errors in mixed setups.
Net effect: The pipeline remains fast, standards-compliant, and ML-friendly—ready for downstream normalization/augmentation while keeping the codebase simple and consistent with prior NumPy-first practices.
Splitting the dataset¶
In [12]:
# Split the dataset (60% train, 20% validation, 20% test)
X_temp, X_test, y_temp, y_test = train_test_split(
images_processed, labels, test_size=0.2, random_state=42, stratify=labels)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)
Data Normalization¶
In [13]:
# Data Normalization
X_train_norm = X_train.astype('float32') / 255.0
X_val_norm = X_val.astype('float32') / 255.0
X_test_norm = X_test.astype('float32') / 255.0
print(f"\nNormalization completed!")
Normalization completed!
Model Building¶
Model Evaluation Criterion¶
Utility Functions¶
In [14]:
def model_performance_classification(model, predictors, target):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
"""
# Predict and apply threshold
pred = model.predict(predictors).reshape(-1) > 0.5
# Convert target to numpy array if it's a pandas Series
if hasattr(target, "to_numpy"):
target = target.to_numpy().reshape(-1)
else:
target = target.reshape(-1)
# Compute metrics
acc = accuracy_score(target, pred)
recall = recall_score(target, pred, average='weighted')
precision = precision_score(target, pred, average='weighted')
f1 = f1_score(target, pred, average='weighted')
# Return as DataFrame
df_perf = pd.DataFrame({
"Accuracy": [acc],
"Recall": [recall],
"Precision": [precision],
"F1 Score": [f1]
})
return df_perf
In [15]:
def plot_confusion_matrix(model, predictors, target, ml=False):
"""
Function to plot the confusion matrix
model: classifier
predictors: independent variables
target: dependent variable
ml: To specify if the model used is an sklearn ML model or not (True means ML model)
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors).reshape(-1) > 0.5
# Ensure compatibility with both pandas Series and numpy array
if hasattr(target, "to_numpy"):
target = target.to_numpy().reshape(-1)
else:
target = target.reshape(-1)
# Plotting the Confusion Matrix using confusion matrix() function which is also predefined tensorflow module
confusion_matrix = tf.math.confusion_matrix(target, pred)
f, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(
confusion_matrix,
annot=True,
linewidths=.4,
fmt="d",
square=True,
ax=ax
)
plt.show()
In [16]:
# defining a function to plot training and validation metrics from a Keras model history
def plot_training_history(history, title="Training History"):
"""
Function to plot training and validation accuracy and loss over epochs
history: Keras History object returned by model.fit()
title: plot title (optional)
"""
# creating subplot with two axes side-by-side
fig, axes = plt.subplots(1, 2, figsize=(15, 5))
# plotting training and validation accuracy
axes[0].plot(history.history['accuracy'], label='Training Accuracy')
axes[0].plot(history.history['val_accuracy'], label='Validation Accuracy')
axes[0].set_title('Model Accuracy') # setting title
axes[0].set_xlabel('Epoch') # setting x-axis label
axes[0].set_ylabel('Accuracy') # setting y-axis label
axes[0].legend() # showing legend
axes[0].grid(True) # adding grid for better readability
# plotting training and validation loss
axes[1].plot(history.history['loss'], label='Training Loss')
axes[1].plot(history.history['val_loss'], label='Validation Loss')
axes[1].set_title('Model Loss') # setting title
axes[1].set_xlabel('Epoch') # setting x-axis label
axes[1].set_ylabel('Loss') # setting y-axis label
axes[1].legend() # showing legend
axes[1].grid(True) # adding grid
# setting the overall title and adjusting layout
plt.suptitle(title, fontsize=16)
plt.tight_layout()
plt.show()
In [17]:
# defining a function to visualize model predictions on sample images
def visualize_predictions(model, X_data, y_data, n_samples=8, title="Model Predictions"):
"""
Function to visualize model predictions on a random subset of images
model: trained Keras model
X_data: image data (NumPy array)
y_data: true labels (Pandas DataFrame or Series)
n_samples: number of samples to display (default = 8)
title: title of the overall plot (optional)
"""
# generate predicted probabilities
y_pred_prob = model.predict(X_data)
# convert probabilities to binary class predictions
y_pred = (y_pred_prob > 0.5).astype(int)
# randomly select sample indices
indices = np.random.choice(len(X_data), n_samples, replace=False)
# create a 2x4 subplot grid
fig, axes = plt.subplots(2, 4, figsize=(16, 8))
axes = axes.ravel() # flatten axes array for easy indexing
# iterate through selected indices
for i, idx in enumerate(indices):
# show grayscale or RGB image depending on channel count
if X_data.shape[-1] == 1: # grayscale image
axes[i].imshow(X_data[idx].squeeze(), cmap='gray')
else: # RGB image
axes[i].imshow(X_data[idx])
# extract labels and confidence
true_val = y_data.iloc[idx, 0]
pred_val = y_pred[idx][0]
true_label = "With Helmet" if true_val == 1 else "Without Helmet"
pred_label = "With Helmet" if pred_val == 1 else "Without Helmet"
confidence = y_pred_prob[idx][0] if pred_val == 1 else 1 - y_pred_prob[idx][0]
# color the title green if prediction is correct, red if incorrect
color = 'green' if true_val == pred_val else 'red'
# set the image title
axes[i].set_title(f'True: {true_label}\nPred: {pred_label}\nConf: {confidence:.2f}',
color=color)
# hide axis
axes[i].axis('off')
# set global plot title and adjust layout
plt.suptitle(title, fontsize=16)
plt.tight_layout()
plt.show()
Model 1: Simple Convolutional Neural Network (CNN)¶
In [18]:
def create_simple_cnn(input_shape):
"""Create a simple CNN model"""
model = Sequential([
# First Convolutional Block
Conv2D(32, (3, 3), activation='relu', input_shape=input_shape),
MaxPooling2D(2, 2),
# Second Convolutional Block
Conv2D(64, (3, 3), activation='relu'),
MaxPooling2D(2, 2),
# Third Convolutional Block
Conv2D(128, (3, 3), activation='relu'),
MaxPooling2D(2, 2),
# Flatten and Dense layers
Flatten(),
Dense(512, activation='relu'),
Dropout(0.5),
Dense(1, activation='sigmoid')
])
return model
# Create and compile the model
input_shape = (X_train_norm.shape[1], X_train_norm.shape[2], X_train_norm.shape[3])
model_cnn = create_simple_cnn(input_shape)
model_cnn.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy']
)
print("Simple CNN Model Architecture:")
model_cnn.summary()
# Define callbacks
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.0001)
# Train the model
print("\nTraining Simple CNN...")
history_cnn = model_cnn.fit(
X_train_norm, y_train,
batch_size=32,
epochs=50,
validation_data=(X_val_norm, y_val),
callbacks=[early_stopping, reduce_lr],
verbose=1
)
Simple CNN Model Architecture:
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d (Conv2D) │ (None, 198, 198, 32) │ 320 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d (MaxPooling2D) │ (None, 99, 99, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_1 (Conv2D) │ (None, 97, 97, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_1 (MaxPooling2D) │ (None, 48, 48, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_2 (Conv2D) │ (None, 46, 46, 128) │ 73,856 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_2 (MaxPooling2D) │ (None, 23, 23, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 67712) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 512) │ 34,669,056 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 513 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 34,762,241 (132.61 MB)
Trainable params: 34,762,241 (132.61 MB)
Non-trainable params: 0 (0.00 B)
Training Simple CNN... Epoch 1/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 29s 1s/step - accuracy: 0.5514 - loss: 1.0173 - val_accuracy: 0.9365 - val_loss: 0.3543 - learning_rate: 0.0010 Epoch 2/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 16s 71ms/step - accuracy: 0.9623 - loss: 0.2620 - val_accuracy: 0.9921 - val_loss: 0.0231 - learning_rate: 0.0010 Epoch 3/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 71ms/step - accuracy: 0.9935 - loss: 0.0300 - val_accuracy: 0.9921 - val_loss: 0.0073 - learning_rate: 0.0010 Epoch 4/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 61ms/step - accuracy: 0.9838 - loss: 0.0467 - val_accuracy: 1.0000 - val_loss: 0.0087 - learning_rate: 0.0010 Epoch 5/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 61ms/step - accuracy: 0.9896 - loss: 0.0434 - val_accuracy: 1.0000 - val_loss: 0.0154 - learning_rate: 0.0010 Epoch 6/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 50ms/step - accuracy: 0.9756 - loss: 0.0799 - val_accuracy: 1.0000 - val_loss: 0.0602 - learning_rate: 0.0010 Epoch 7/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 53ms/step - accuracy: 0.9978 - loss: 0.0545 - val_accuracy: 0.9921 - val_loss: 0.0076 - learning_rate: 0.0010 Epoch 8/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 64ms/step - accuracy: 0.9891 - loss: 0.0207 - val_accuracy: 1.0000 - val_loss: 0.0044 - learning_rate: 0.0010 Epoch 9/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 63ms/step - accuracy: 0.9982 - loss: 0.0104 - val_accuracy: 1.0000 - val_loss: 5.4646e-04 - learning_rate: 0.0010 Epoch 10/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 1.0000 - loss: 0.0015 - val_accuracy: 1.0000 - val_loss: 9.0589e-04 - learning_rate: 0.0010 Epoch 11/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 64ms/step - accuracy: 1.0000 - loss: 0.0013 - val_accuracy: 1.0000 - val_loss: 7.0463e-05 - learning_rate: 0.0010 Epoch 12/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.9982 - loss: 0.0025 - val_accuracy: 0.9921 - val_loss: 0.0113 - learning_rate: 0.0010 Epoch 13/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.9923 - loss: 0.0335 - val_accuracy: 1.0000 - val_loss: 0.0013 - learning_rate: 0.0010 Epoch 14/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 50ms/step - accuracy: 0.9986 - loss: 0.0042 - val_accuracy: 1.0000 - val_loss: 0.0021 - learning_rate: 0.0010 Epoch 15/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.9978 - loss: 0.0084 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 0.0010 Epoch 16/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.9977 - loss: 0.0076 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 0.0010 Epoch 17/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 54ms/step - accuracy: 1.0000 - loss: 0.0014 - val_accuracy: 1.0000 - val_loss: 0.0014 - learning_rate: 2.0000e-04 Epoch 18/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 55ms/step - accuracy: 1.0000 - loss: 0.0018 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 2.0000e-04 Epoch 19/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 53ms/step - accuracy: 1.0000 - loss: 0.0011 - val_accuracy: 1.0000 - val_loss: 9.9570e-04 - learning_rate: 2.0000e-04 Epoch 20/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 1.0000 - loss: 0.0012 - val_accuracy: 1.0000 - val_loss: 7.9754e-04 - learning_rate: 2.0000e-04 Epoch 21/50 12/12 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 1.0000 - loss: 0.0010 - val_accuracy: 1.0000 - val_loss: 6.3857e-04 - learning_rate: 2.0000e-04
In [19]:
# Plot training history
plot_training_history(history_cnn, "Simple CNN Training History")
In [20]:
# Evaluate performance
print("\nSimple CNN Performance on Validation Set:")
perf_cnn_val = model_performance_classification(model_cnn, X_val_norm, y_val)
print(perf_cnn_val)
Simple CNN Performance on Validation Set: 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 121ms/step Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
In [21]:
# Plot confusion matrix
plot_confusion_matrix(model_cnn, X_val_norm, y_val, "Simple CNN - Validation Confusion Matrix")
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step
Observations from Model Results – Simple CNN Model¶
✅ Findings from the Simple CNN Run¶
🏗️ Model at a Glance¶
- The network stacks three Conv→ReLU→MaxPool blocks with filters 32 → 64 → 128, then Flatten → Dense(512) → Dropout(0.5) → Sigmoid.
- Param count: ~34.76M total, all trainable.
- Where the bulk lives: The Flatten → Dense(512) section dominates the parameter budget (~34.6M), making this the main source of capacity and potential overfitting.
Tip: If you want to shrink parameters without losing much accuracy, consider replacing
Flatten()withGlobalAveragePooling2D()and/or reducing the dense width, and add L2 weight decay.
📈 Training Dynamics¶
- Accuracy: Training and validation accuracy climb to ~100% within a few epochs.
- Loss: Both training and validation loss fall rapidly and plateau near zero, indicating highly confident predictions.
What this suggests
- The task/data split appears learnable and clean, or the signal (helmet vs. no-helmet) is very strong.
- The tight tracking between train and validation curves implies little observable overfitting under the used callbacks (EarlyStopping + ReduceLROnPlateau).
Sanity checks worth running: confirm no data leakage/duplication across train–val, and verify that augmentations aren’t applied inconsistently.
🔢 Validation Confusion Matrix (your run)¶
- Class 0 (Without Helmet): 64/64 correct
- Class 1 (With Helmet): 62/62 correct
- Errors: None observed (no FPs/FNs)
Implication
- On this validation split, the model delivers perfect scores (accuracy/precision/recall/F1 = 100%).
- When results are this high, it’s prudent to cross-check with a stratified k-fold or a held-out test set to ensure robustness.
Vizualizing the predictions¶
In [22]:
visualize_predictions(model_cnn, X_val_norm, y_val, title="Simple CNN - Sample Predictions")
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step
SIMPLE CNN MODEL ANALYSIS
- The current CNN stack (3 conv–pool blocks → Dense(512) → sigmoid) cleanly separates the two classes on the validation split, yielding near-perfect metrics.
- Such “perfect” validation results can be fragile; they may not hold under domain shift or noisier inputs. Strongly recommend confirming on a held-out test set and/or stratified k-fold CV.
- Run leakage checks (no duplicate or near-duplicate images across train/val; identical preprocessing; labels aligned).
- Stress-test generalization with heavier augmentations (brightness/contrast, small rotations/crops) and, if possible, evaluate on out-of-distribution samples (different cameras/sites).
- Consider parameter-efficiency and regularization: swap
FlattenforGlobalAveragePooling2D, reduce the dense width, add L2 weight decay, and keep Dropout. - Go beyond accuracy: inspect confusion matrix, precision/recall, ROC–AUC, and probability calibration; optionally use saliency/Grad-CAM to verify the model focuses on helmets, not background cues.
Model 2: (VGG-16 (Base))¶
In [23]:
def create_vgg16_base(input_shape):
"""Create VGG-16 base model"""
# For grayscale input, we need to convert to 3 channels
if input_shape[-1] == 1:
inputs = tf.keras.Input(shape=input_shape)
# Resize to 224x224
x = tf.keras.layers.Resizing(224, 224)(inputs)
# Convert grayscale (1 channel) to 3 channels
x = tf.keras.layers.Conv2D(3, (1, 1), activation='linear')(x)
# Load VGG16 base
vgg_base = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
vgg_base.trainable = False
x = vgg_base(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
else:
inputs = tf.keras.Input(shape=input_shape)
# Resize to 224x224
x = tf.keras.layers.Resizing(224, 224)(inputs)
# If already RGB, usar diretamente
vgg_base = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
vgg_base.trainable = False
x = vgg_base(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
return model
# Create and compile VGG-16 base model
model_vgg_base = create_vgg16_base(input_shape)
model_vgg_base.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy']
)
print("VGG-16 Base Model Architecture:")
model_vgg_base.summary()
# Train the model
print("\nTraining VGG-16 Base Model...")
history_vgg_base = model_vgg_base.fit(
X_train_norm, y_train,
batch_size=32,
epochs=30,
validation_data=(X_val_norm, y_val),
callbacks=[early_stopping, reduce_lr],
verbose=1
)
VGG-16 Base Model Architecture:
Model: "functional_1"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_1 (InputLayer) │ (None, 200, 200, 1) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ resizing (Resizing) │ (None, 224, 224, 1) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_3 (Conv2D) │ (None, 224, 224, 3) │ 6 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ vgg16 (Functional) │ (None, 7, 7, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 513 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 14,715,207 (56.13 MB)
Trainable params: 519 (2.03 KB)
Non-trainable params: 14,714,688 (56.13 MB)
Training VGG-16 Base Model... Epoch 1/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 85s 5s/step - accuracy: 0.5115 - loss: 0.7140 - val_accuracy: 0.7460 - val_loss: 0.6491 - learning_rate: 0.0010 Epoch 2/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 63s 386ms/step - accuracy: 0.7808 - loss: 0.6474 - val_accuracy: 0.9841 - val_loss: 0.5892 - learning_rate: 0.0010 Epoch 3/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 442ms/step - accuracy: 0.9394 - loss: 0.5925 - val_accuracy: 0.9841 - val_loss: 0.5347 - learning_rate: 0.0010 Epoch 4/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 389ms/step - accuracy: 0.9493 - loss: 0.5422 - val_accuracy: 0.9841 - val_loss: 0.4852 - learning_rate: 0.0010 Epoch 5/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 394ms/step - accuracy: 0.9726 - loss: 0.4955 - val_accuracy: 0.9841 - val_loss: 0.4407 - learning_rate: 0.0010 Epoch 6/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 402ms/step - accuracy: 0.9813 - loss: 0.4528 - val_accuracy: 0.9841 - val_loss: 0.4007 - learning_rate: 0.0010 Epoch 7/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 404ms/step - accuracy: 0.9937 - loss: 0.4143 - val_accuracy: 0.9841 - val_loss: 0.3651 - learning_rate: 0.0010 Epoch 8/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 459ms/step - accuracy: 0.9937 - loss: 0.3797 - val_accuracy: 0.9841 - val_loss: 0.3336 - learning_rate: 0.0010 Epoch 9/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 446ms/step - accuracy: 0.9946 - loss: 0.3488 - val_accuracy: 0.9841 - val_loss: 0.3058 - learning_rate: 0.0010 Epoch 10/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 384ms/step - accuracy: 0.9946 - loss: 0.3212 - val_accuracy: 0.9841 - val_loss: 0.2813 - learning_rate: 0.0010 Epoch 11/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 445ms/step - accuracy: 0.9946 - loss: 0.2966 - val_accuracy: 0.9841 - val_loss: 0.2596 - learning_rate: 0.0010 Epoch 12/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 441ms/step - accuracy: 0.9946 - loss: 0.2746 - val_accuracy: 0.9841 - val_loss: 0.2403 - learning_rate: 0.0010 Epoch 13/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 384ms/step - accuracy: 0.9967 - loss: 0.2549 - val_accuracy: 0.9841 - val_loss: 0.2232 - learning_rate: 0.0010 Epoch 14/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 389ms/step - accuracy: 0.9967 - loss: 0.2372 - val_accuracy: 0.9841 - val_loss: 0.2079 - learning_rate: 0.0010 Epoch 15/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 446ms/step - accuracy: 1.0000 - loss: 0.2212 - val_accuracy: 0.9841 - val_loss: 0.1941 - learning_rate: 0.0010 Epoch 16/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 449ms/step - accuracy: 1.0000 - loss: 0.2068 - val_accuracy: 0.9841 - val_loss: 0.1818 - learning_rate: 0.0010 Epoch 17/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 448ms/step - accuracy: 1.0000 - loss: 0.1937 - val_accuracy: 0.9841 - val_loss: 0.1705 - learning_rate: 0.0010 Epoch 18/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 448ms/step - accuracy: 1.0000 - loss: 0.1818 - val_accuracy: 0.9841 - val_loss: 0.1603 - learning_rate: 0.0010 Epoch 19/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 391ms/step - accuracy: 1.0000 - loss: 0.1710 - val_accuracy: 0.9841 - val_loss: 0.1510 - learning_rate: 0.0010 Epoch 20/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 395ms/step - accuracy: 1.0000 - loss: 0.1610 - val_accuracy: 0.9841 - val_loss: 0.1425 - learning_rate: 0.0010 Epoch 21/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 393ms/step - accuracy: 1.0000 - loss: 0.1518 - val_accuracy: 0.9921 - val_loss: 0.1347 - learning_rate: 0.0010 Epoch 22/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 449ms/step - accuracy: 1.0000 - loss: 0.1435 - val_accuracy: 0.9921 - val_loss: 0.1277 - learning_rate: 0.0010 Epoch 23/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 388ms/step - accuracy: 1.0000 - loss: 0.1358 - val_accuracy: 0.9921 - val_loss: 0.1213 - learning_rate: 0.0010 Epoch 24/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 448ms/step - accuracy: 1.0000 - loss: 0.1288 - val_accuracy: 0.9921 - val_loss: 0.1154 - learning_rate: 0.0010 Epoch 25/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 444ms/step - accuracy: 1.0000 - loss: 0.1224 - val_accuracy: 0.9921 - val_loss: 0.1100 - learning_rate: 0.0010 Epoch 26/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 384ms/step - accuracy: 1.0000 - loss: 0.1164 - val_accuracy: 1.0000 - val_loss: 0.1049 - learning_rate: 0.0010 Epoch 27/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 448ms/step - accuracy: 1.0000 - loss: 0.1109 - val_accuracy: 1.0000 - val_loss: 0.1003 - learning_rate: 0.0010 Epoch 28/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 447ms/step - accuracy: 1.0000 - loss: 0.1058 - val_accuracy: 1.0000 - val_loss: 0.0959 - learning_rate: 0.0010 Epoch 29/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 388ms/step - accuracy: 1.0000 - loss: 0.1010 - val_accuracy: 1.0000 - val_loss: 0.0919 - learning_rate: 0.0010 Epoch 30/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 394ms/step - accuracy: 1.0000 - loss: 0.0965 - val_accuracy: 1.0000 - val_loss: 0.0882 - learning_rate: 0.0010
In [24]:
# Plot training history
plot_training_history(history_vgg_base, "VGG-16 Base Training History")
In [25]:
# Evaluate performance
print("\nVGG-16 Base Performance on Validation Set:")
perf_vgg_base_val = model_performance_classification(model_vgg_base, X_val_norm, y_val)
print(perf_vgg_base_val)
VGG-16 Base Performance on Validation Set: 4/4 ━━━━━━━━━━━━━━━━━━━━ 2s 388ms/step Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
In [26]:
# Plot confusion matrix
plot_confusion_matrix(model_vgg_base, X_val_norm, y_val, "VGG-16 Base - Validation Confusion Matrix")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 162ms/step
Observations from Model Results – VGG-16 Base Model¶
🧱 Architecture Snapshot¶
- Inputs are resized to 224×224×3 to match VGG’s expected shape.
- A 1×1 conv expands grayscale inputs to 3 channels.
- The VGG-16 backbone (ImageNet) is frozen; only the lightweight head is trainable.
- Head: Global Average Pooling → Dense(sigmoid) for binary output.
Parameters
- Total: ~14.7M
- Trainable: ~519 (primarily the 1×1 conv + final dense layer)
- Implication: Most capacity sits in the frozen VGG stack, leveraging pretrained features and lowering overfitting risk.
📈 Training Behavior¶
- Accuracy: Train/validation accuracy rise smoothly, surpassing 99% around epoch ~20.
- Loss: Both curves decrease steadily without spikes, indicating stable optimization.
Interpretation
- Pretrained VGG-16 features transfer effectively to this task.
- Train/val curves track closely, suggesting good generalization with the current freeze strategy and callbacks.
✅ Validation Confusion Matrix (Binary)¶
- Class 0 (Without Helmet): 64 correct
- Class 1 (With Helmet): 61 correct
- Misclassifications: 1 (a false negative for class 1; no false positives for class 1)
Metrics (approx.)
- Accuracy: ~99.2%
- For class 1 (positive): Precision ~100%, Recall ~98.4% (1 FN)
- For class 0: Recall ~100%, Precision ~98.5% (due to the single class-1→class-0 error)
Visualizing the prediction:¶
In [27]:
# Visualize predictions
visualize_predictions(model_vgg_base, X_val_norm, y_val, title="VGG-16 Base - Sample Predictions")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 163ms/step
VGG-16 BASE MODEL ANALYSIS
- The frozen VGG-16 backbone with a light classification head delivered near-perfect validation performance, reliably separating “helmet” vs “no helmet” images.
- Thanks to transfer learning, the model trains only a small number of weights while reusing rich, pretrained features, achieving strong results with minimal trainable parameters.
- Despite the excellent validation metrics, it’s important to verify generalization on a held-out test set (and/or cross-validation), especially for noisier or out-of-distribution data.
Model 3: (VGG-16 (Base + FFNN))¶
In [28]:
def create_vgg16_ffnn(input_shape):
"""Create VGG-16 with enhanced FFNN head"""
if input_shape[-1] == 1:
# Grayscale input: resize and convert to 3 channels
inputs = tf.keras.Input(shape=input_shape)
x = tf.keras.layers.Resizing(224, 224)(inputs)
x = tf.keras.layers.Conv2D(3, (1, 1), activation='linear')(x)
# Load VGG16 base
vgg_base = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
vgg_base.trainable = False
x = vgg_base(x)
# Enhanced FFNN head
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(256, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.2)(x)
outputs = Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
else:
# RGB input: resize handled internally
inputs = tf.keras.Input(shape=input_shape)
x = tf.keras.layers.Resizing(224, 224)(inputs)
vgg_base = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
vgg_base.trainable = False
x = vgg_base(x)
# Enhanced FFNN head
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(256, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.2)(x)
outputs = Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
return model
# Create and compile VGG-16 FFNN model
model_vgg_ffnn = create_vgg16_ffnn(input_shape)
model_vgg_ffnn.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy']
)
print("VGG-16 FFNN Model Architecture:")
model_vgg_ffnn.summary()
# Train the model
print("\nTraining VGG-16 FFNN Model...")
history_vgg_ffnn = model_vgg_ffnn.fit(
X_train_norm, y_train,
batch_size=32,
epochs=40,
validation_data=(X_val_norm, y_val),
callbacks=[early_stopping, reduce_lr],
verbose=1
)
VGG-16 FFNN Model Architecture:
Model: "functional_2"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_3 (InputLayer) │ (None, 200, 200, 1) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ resizing_1 (Resizing) │ (None, 224, 224, 1) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_4 (Conv2D) │ (None, 224, 224, 3) │ 6 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ vgg16 (Functional) │ (None, 7, 7, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d_1 │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_3 (Dense) │ (None, 512) │ 262,656 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization │ (None, 512) │ 2,048 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_4 (Dense) │ (None, 256) │ 131,328 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_1 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_2 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_5 (Dense) │ (None, 128) │ 32,896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_3 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_6 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 15,144,775 (57.77 MB)
Trainable params: 428,551 (1.63 MB)
Non-trainable params: 14,716,224 (56.14 MB)
Training VGG-16 FFNN Model... Epoch 1/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 20s 1s/step - accuracy: 0.7507 - loss: 0.4766 - val_accuracy: 0.9444 - val_loss: 0.4092 - learning_rate: 0.0010 Epoch 2/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 8s 388ms/step - accuracy: 0.9910 - loss: 0.0303 - val_accuracy: 0.9524 - val_loss: 0.3104 - learning_rate: 0.0010 Epoch 3/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 395ms/step - accuracy: 0.9978 - loss: 0.0141 - val_accuracy: 0.9841 - val_loss: 0.2568 - learning_rate: 0.0010 Epoch 4/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 394ms/step - accuracy: 0.9978 - loss: 0.0071 - val_accuracy: 0.9841 - val_loss: 0.2087 - learning_rate: 0.0010 Epoch 5/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 399ms/step - accuracy: 1.0000 - loss: 0.0100 - val_accuracy: 1.0000 - val_loss: 0.1670 - learning_rate: 0.0010 Epoch 6/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 454ms/step - accuracy: 1.0000 - loss: 0.0045 - val_accuracy: 1.0000 - val_loss: 0.1217 - learning_rate: 0.0010 Epoch 7/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 452ms/step - accuracy: 1.0000 - loss: 0.0041 - val_accuracy: 1.0000 - val_loss: 0.1005 - learning_rate: 0.0010 Epoch 8/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 387ms/step - accuracy: 0.9967 - loss: 0.0036 - val_accuracy: 1.0000 - val_loss: 0.0839 - learning_rate: 0.0010 Epoch 9/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 388ms/step - accuracy: 1.0000 - loss: 0.0025 - val_accuracy: 1.0000 - val_loss: 0.0759 - learning_rate: 0.0010 Epoch 10/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 386ms/step - accuracy: 1.0000 - loss: 0.0016 - val_accuracy: 1.0000 - val_loss: 0.0776 - learning_rate: 0.0010 Epoch 11/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 385ms/step - accuracy: 1.0000 - loss: 0.0019 - val_accuracy: 0.9921 - val_loss: 0.0785 - learning_rate: 0.0010 Epoch 12/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 393ms/step - accuracy: 1.0000 - loss: 6.9030e-04 - val_accuracy: 0.9921 - val_loss: 0.0716 - learning_rate: 0.0010 Epoch 13/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 444ms/step - accuracy: 1.0000 - loss: 9.4059e-04 - val_accuracy: 0.9921 - val_loss: 0.0607 - learning_rate: 0.0010 Epoch 14/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 445ms/step - accuracy: 1.0000 - loss: 9.5406e-04 - val_accuracy: 0.9921 - val_loss: 0.0535 - learning_rate: 0.0010 Epoch 15/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 384ms/step - accuracy: 1.0000 - loss: 0.0011 - val_accuracy: 0.9921 - val_loss: 0.0406 - learning_rate: 0.0010 Epoch 16/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 446ms/step - accuracy: 1.0000 - loss: 3.4030e-04 - val_accuracy: 1.0000 - val_loss: 0.0310 - learning_rate: 0.0010 Epoch 17/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 391ms/step - accuracy: 1.0000 - loss: 6.8016e-04 - val_accuracy: 0.9921 - val_loss: 0.0296 - learning_rate: 0.0010 Epoch 18/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 391ms/step - accuracy: 1.0000 - loss: 3.6946e-04 - val_accuracy: 1.0000 - val_loss: 0.0239 - learning_rate: 0.0010 Epoch 19/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 394ms/step - accuracy: 1.0000 - loss: 6.0061e-04 - val_accuracy: 1.0000 - val_loss: 0.0185 - learning_rate: 0.0010 Epoch 20/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 394ms/step - accuracy: 1.0000 - loss: 3.6359e-04 - val_accuracy: 1.0000 - val_loss: 0.0180 - learning_rate: 0.0010 Epoch 21/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 400ms/step - accuracy: 1.0000 - loss: 5.3594e-04 - val_accuracy: 1.0000 - val_loss: 0.0153 - learning_rate: 0.0010 Epoch 22/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 450ms/step - accuracy: 1.0000 - loss: 1.7851e-04 - val_accuracy: 1.0000 - val_loss: 0.0130 - learning_rate: 0.0010 Epoch 23/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 389ms/step - accuracy: 1.0000 - loss: 1.8403e-04 - val_accuracy: 1.0000 - val_loss: 0.0111 - learning_rate: 0.0010 Epoch 24/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 385ms/step - accuracy: 1.0000 - loss: 5.6694e-04 - val_accuracy: 0.9921 - val_loss: 0.0151 - learning_rate: 0.0010 Epoch 25/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 441ms/step - accuracy: 1.0000 - loss: 2.3822e-04 - val_accuracy: 0.9841 - val_loss: 0.0689 - learning_rate: 0.0010 Epoch 26/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 380ms/step - accuracy: 0.9978 - loss: 0.0036 - val_accuracy: 0.9921 - val_loss: 0.0115 - learning_rate: 0.0010 Epoch 27/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 385ms/step - accuracy: 1.0000 - loss: 8.5265e-04 - val_accuracy: 0.9921 - val_loss: 0.0151 - learning_rate: 0.0010 Epoch 28/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 441ms/step - accuracy: 1.0000 - loss: 6.2998e-04 - val_accuracy: 0.9603 - val_loss: 0.0863 - learning_rate: 0.0010 Epoch 29/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 387ms/step - accuracy: 1.0000 - loss: 9.5228e-04 - val_accuracy: 0.9841 - val_loss: 0.0716 - learning_rate: 2.0000e-04 Epoch 30/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 390ms/step - accuracy: 1.0000 - loss: 9.0208e-04 - val_accuracy: 0.9841 - val_loss: 0.0374 - learning_rate: 2.0000e-04 Epoch 31/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 390ms/step - accuracy: 1.0000 - loss: 3.5794e-04 - val_accuracy: 0.9921 - val_loss: 0.0198 - learning_rate: 2.0000e-04 Epoch 32/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 446ms/step - accuracy: 1.0000 - loss: 4.0517e-04 - val_accuracy: 0.9921 - val_loss: 0.0120 - learning_rate: 2.0000e-04 Epoch 33/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 448ms/step - accuracy: 1.0000 - loss: 2.6809e-04 - val_accuracy: 0.9921 - val_loss: 0.0080 - learning_rate: 2.0000e-04 Epoch 34/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 392ms/step - accuracy: 1.0000 - loss: 1.4853e-04 - val_accuracy: 1.0000 - val_loss: 0.0054 - learning_rate: 2.0000e-04 Epoch 35/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 390ms/step - accuracy: 1.0000 - loss: 1.8227e-04 - val_accuracy: 1.0000 - val_loss: 0.0034 - learning_rate: 2.0000e-04 Epoch 36/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 392ms/step - accuracy: 1.0000 - loss: 3.9966e-04 - val_accuracy: 1.0000 - val_loss: 0.0018 - learning_rate: 2.0000e-04 Epoch 37/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 449ms/step - accuracy: 1.0000 - loss: 4.1796e-04 - val_accuracy: 1.0000 - val_loss: 0.0011 - learning_rate: 2.0000e-04 Epoch 38/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 446ms/step - accuracy: 1.0000 - loss: 3.0393e-04 - val_accuracy: 1.0000 - val_loss: 6.1779e-04 - learning_rate: 2.0000e-04 Epoch 39/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 10s 386ms/step - accuracy: 1.0000 - loss: 8.0061e-05 - val_accuracy: 1.0000 - val_loss: 4.0557e-04 - learning_rate: 2.0000e-04 Epoch 40/40 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 391ms/step - accuracy: 1.0000 - loss: 1.4105e-04 - val_accuracy: 1.0000 - val_loss: 2.8944e-04 - learning_rate: 2.0000e-04
In [29]:
# Plot training history
plot_training_history(history_vgg_ffnn, "VGG-16 FFNN Training History")
In [30]:
# Evaluate performance
print("\nVGG-16 FFNN Performance on Validation Set:")
perf_vgg_ffnn_val = model_performance_classification(model_vgg_ffnn, X_val_norm, y_val)
print(perf_vgg_ffnn_val)
VGG-16 FFNN Performance on Validation Set: 4/4 ━━━━━━━━━━━━━━━━━━━━ 2s 419ms/step Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
In [31]:
# Plot confusion matrix
plot_confusion_matrix(model_vgg_ffnn, X_val_norm, y_val, "VGG-16 FFNN - Validation Confusion Matrix")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 161ms/step
🔎 Observations — VGG-16 + FFNN Head¶
🧱 Architecture Summary¶
- Inputs are resized to 224×224; a 1×1 conv lifts grayscale to 3 channels.
- VGG-16 (ImageNet) acts as a frozen feature extractor.
- Head: Global Average Pooling → Dense(512) → Dense(256) → Dense(128) with BatchNorm and Dropout, then a sigmoid output.
- Params: ~15.1M total; ~428k trainable (head); ~14.7M non-trainable (VGG backbone).
- Note: The design balances strong pretrained features with a richer, task-specific classifier.
📈 Training Behavior¶
- Accuracy: Train and validation climb past 99% and stabilize at ~100%.
- Loss: Both curves decrease smoothly with no visible divergence.
- Interpretation: The frozen VGG-16 supplies robust features while the regularized head captures task nuances; close train/val tracking suggests good validation-set generalization.
✅ Validation Confusion Matrix¶
- Class 0 (Without Helmet): 64/64 correct
- Class 1 (With Helmet): 62/62 correct
- Errors: None observed (no FP/FN)
Implication: On this split, metrics reach 100% (accuracy, precision, recall, F1). Results look excellent, but confirm on a held-out test set and/or cross-validation to ensure robustness to new or noisier data.
🧭 Practical Notes¶
- Double-check for data leakage/duplication across splits.
- Evaluate under domain shift (different sites/cameras) and consider augmentation to stress-test.
- Optionally unfreeze top VGG blocks for a brief fine-tune if you need extra robustness.
Visualizing the predictions¶
In [32]:
# Visualize predictions
visualize_predictions(model_vgg_ffnn, X_val_norm, y_val, title="VGG-16 FFNN - Sample Predictions")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 163ms/step
VGG-16 + FFNN MODEL ANALYSIS:
The VGG-16 model with an enhanced feedforward head achieved perfect classification performance on the validation set, accurately distinguishing between images with and without helmets.
By combining transfer learning with a deeper and regularized dense head, the model was able to learn more complex decision boundaries while still benefiting from the pretrained VGG-16 features.
Despite the flawless validation results, further evaluation on an independent test set is essential to verify the model’s ability to generalize to new or noisier data.
Model 4: (VGG-16 (Base + FFNN + Data Augmentation)¶
In most of the real-world case studies, it is challenging to acquire a large number of images and then train CNNs.
To overcome this problem, one approach we might consider is Data Augmentation.
CNNs have the property of translational invariance, which means they can recognise an object even if its appearance shifts translationally in some way. - Taking this attribute into account, we can augment the images using the techniques listed below
- Horizontal Flip (should be set to True/False)
- Vertical Flip (should be set to True/False)
- Height Shift (should be between 0 and 1)
- Width Shift (should be between 0 and 1)
- Rotation (should be between 0 and 180)
- Shear (should be between 0 and 1)
- Zoom (should be between 0 and 1) etc.
Remember, data augmentation should not be used in the validation/test data set.
In [33]:
# Data augmentation configuration for training
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
# Validation generator (no augmentation, only rescaling)
val_datagen = ImageDataGenerator(rescale=1./255)
# Convert grayscale back to RGB format (if necessary)
if X_train.shape[-1] == 1:
X_train_rgb = np.repeat(X_train, 3, axis=-1)
X_val_rgb = np.repeat(X_val, 3, axis=-1)
X_test_rgb = np.repeat(X_test, 3, axis=-1)
else:
X_train_rgb = X_train.copy()
X_val_rgb = X_val.copy()
X_test_rgb = X_test.copy()
# Convert labels to numpy arrays (fix KeyError issue)
y_train_np = np.array(y_train)
y_val_np = np.array(y_val)
y_test_np = np.array(y_test)
print(f"RGB Training data shape: {X_train_rgb.shape}")
# Create data generators
train_generator = train_datagen.flow(X_train_rgb, y_train_np, batch_size=32)
val_generator = val_datagen.flow(X_val_rgb, y_val_np, batch_size=32)
# Display some augmented samples
print("Displaying data augmentation examples...")
fig, axes = plt.subplots(2, 8, figsize=(20, 6))
# Get a batch of augmented images
sample_batch = next(train_generator)
sample_images, sample_labels = sample_batch
for i in range(8):
# Original
axes[0, i].imshow(X_train_rgb[i])
axes[0, i].set_title(f'Original\nLabel: {"Helmet" if y_train_np[i]==1 else "No Helmet"}')
axes[0, i].axis('off')
# Augmented (fix: scale back from [0,1] to [0,255])
axes[1, i].imshow((sample_images[i] * 255).astype('uint8'))
axes[1, i].set_title(f'Augmented\nLabel: {"Helmet" if sample_labels[i]==1 else "No Helmet"}')
axes[1, i].axis('off')
plt.suptitle('Data Augmentation Examples', fontsize=16, fontweight='bold')
plt.tight_layout()
plt.show()
def create_vgg16_augmented():
"""Create VGG-16 model for use with data augmentation and resizing"""
# Define input layer with shape of your current data
inputs = tf.keras.Input(shape=(X_train_rgb.shape[1], X_train_rgb.shape[2], X_train_rgb.shape[3]))
# Resize to 224x224 (required by VGG-16)
x = tf.keras.layers.Resizing(224, 224)(inputs)
# Load VGG16 base
vgg_base = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
vgg_base.trainable = True
for layer in vgg_base.layers[:-4]:
layer.trainable = False
# Pass resized input through VGG base
x = vgg_base(x)
x = GlobalAveragePooling2D()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(256, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.2)(x)
outputs = Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs, name="VGG16_Augmented")
return model
# Create and compile the augmented model
print("Creating VGG-16 with Data Augmentation...")
model_vgg_aug = create_vgg16_augmented()
model_vgg_aug.compile(
optimizer=Adam(learning_rate=0.0001), # Lower learning rate for fine-tuning
loss='binary_crossentropy',
metrics=['accuracy']
)
print("VGG-16 Augmented Model Architecture:")
model_vgg_aug.summary()
# Train the model with data augmentation
print("\nTraining VGG-16 with Data Augmentation...")
history_vgg_aug = model_vgg_aug.fit(
train_generator,
steps_per_epoch=len(X_train_rgb) // 32,
epochs=50,
validation_data=val_generator,
validation_steps=len(X_val_rgb) // 32,
callbacks=[early_stopping, reduce_lr],
verbose=1
)
RGB Training data shape: (378, 200, 200, 3) Displaying data augmentation examples...
WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [1.983..202.513]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [2.527..237.749]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [8.176..250.845]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.0..255.0]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [4.45..205.19]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [27.744..250.228]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.0..255.0]. WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [14.563..214.572].
Creating VGG-16 with Data Augmentation... VGG-16 Augmented Model Architecture:
Model: "VGG16_Augmented"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_5 (InputLayer) │ (None, 200, 200, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ resizing_2 (Resizing) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ vgg16 (Functional) │ (None, 7, 7, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d_2 │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_7 (Dense) │ (None, 512) │ 262,656 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_2 │ (None, 512) │ 2,048 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_4 (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_8 (Dense) │ (None, 256) │ 131,328 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_3 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_5 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_9 (Dense) │ (None, 128) │ 32,896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_6 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_10 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 15,144,769 (57.77 MB)
Trainable params: 7,507,969 (28.64 MB)
Non-trainable params: 7,636,800 (29.13 MB)
Training VGG-16 with Data Augmentation... Epoch 1/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 27s 2s/step - accuracy: 0.7324 - loss: 0.4980 - val_accuracy: 0.8750 - val_loss: 0.5060 - learning_rate: 1.0000e-04 Epoch 2/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 59ms/step - accuracy: 1.0000 - loss: 0.0797 - val_accuracy: 0.9167 - val_loss: 0.4877 - learning_rate: 1.0000e-04 Epoch 3/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 414ms/step - accuracy: 0.9956 - loss: 0.0933 - val_accuracy: 0.9896 - val_loss: 0.3181 - learning_rate: 1.0000e-04 Epoch 4/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 58ms/step - accuracy: 1.0000 - loss: 0.0427 - val_accuracy: 0.9896 - val_loss: 0.2993 - learning_rate: 1.0000e-04 Epoch 5/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 492ms/step - accuracy: 0.9911 - loss: 0.0438 - val_accuracy: 1.0000 - val_loss: 0.2148 - learning_rate: 1.0000e-04 Epoch 6/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 0.9688 - loss: 0.0826 - val_accuracy: 1.0000 - val_loss: 0.2162 - learning_rate: 1.0000e-04 Epoch 7/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 575ms/step - accuracy: 0.9962 - loss: 0.0224 - val_accuracy: 1.0000 - val_loss: 0.1685 - learning_rate: 1.0000e-04 Epoch 8/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 102ms/step - accuracy: 1.0000 - loss: 0.0121 - val_accuracy: 1.0000 - val_loss: 0.1597 - learning_rate: 1.0000e-04 Epoch 9/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 4s 396ms/step - accuracy: 1.0000 - loss: 0.0291 - val_accuracy: 1.0000 - val_loss: 0.1264 - learning_rate: 1.0000e-04 Epoch 10/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 55ms/step - accuracy: 1.0000 - loss: 0.0080 - val_accuracy: 1.0000 - val_loss: 0.1167 - learning_rate: 1.0000e-04 Epoch 11/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 4s 386ms/step - accuracy: 1.0000 - loss: 0.0112 - val_accuracy: 1.0000 - val_loss: 0.0902 - learning_rate: 1.0000e-04 Epoch 12/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 54ms/step - accuracy: 1.0000 - loss: 0.0051 - val_accuracy: 1.0000 - val_loss: 0.0926 - learning_rate: 1.0000e-04 Epoch 13/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 970ms/step - accuracy: 1.0000 - loss: 0.0099 - val_accuracy: 1.0000 - val_loss: 0.0751 - learning_rate: 1.0000e-04 Epoch 14/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 55ms/step - accuracy: 1.0000 - loss: 0.0077 - val_accuracy: 1.0000 - val_loss: 0.0714 - learning_rate: 1.0000e-04 Epoch 15/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 471ms/step - accuracy: 1.0000 - loss: 0.0076 - val_accuracy: 1.0000 - val_loss: 0.0532 - learning_rate: 1.0000e-04 Epoch 16/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 53ms/step - accuracy: 1.0000 - loss: 0.0099 - val_accuracy: 1.0000 - val_loss: 0.0563 - learning_rate: 1.0000e-04 Epoch 17/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 9s 382ms/step - accuracy: 1.0000 - loss: 0.0055 - val_accuracy: 1.0000 - val_loss: 0.0442 - learning_rate: 1.0000e-04 Epoch 18/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 1.0000 - loss: 0.0247 - val_accuracy: 1.0000 - val_loss: 0.0463 - learning_rate: 1.0000e-04 Epoch 19/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 483ms/step - accuracy: 1.0000 - loss: 0.0093 - val_accuracy: 1.0000 - val_loss: 0.0344 - learning_rate: 1.0000e-04 Epoch 20/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 110ms/step - accuracy: 1.0000 - loss: 0.0013 - val_accuracy: 1.0000 - val_loss: 0.0324 - learning_rate: 1.0000e-04 Epoch 21/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 464ms/step - accuracy: 1.0000 - loss: 0.0042 - val_accuracy: 1.0000 - val_loss: 0.0242 - learning_rate: 1.0000e-04 Epoch 22/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 56ms/step - accuracy: 1.0000 - loss: 0.0038 - val_accuracy: 1.0000 - val_loss: 0.0228 - learning_rate: 1.0000e-04 Epoch 23/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 9s 377ms/step - accuracy: 1.0000 - loss: 0.0079 - val_accuracy: 1.0000 - val_loss: 0.0187 - learning_rate: 1.0000e-04 Epoch 24/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 104ms/step - accuracy: 1.0000 - loss: 0.0029 - val_accuracy: 1.0000 - val_loss: 0.0192 - learning_rate: 1.0000e-04 Epoch 25/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 9s 382ms/step - accuracy: 1.0000 - loss: 0.0034 - val_accuracy: 1.0000 - val_loss: 0.0165 - learning_rate: 1.0000e-04 Epoch 26/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 108ms/step - accuracy: 1.0000 - loss: 0.0027 - val_accuracy: 1.0000 - val_loss: 0.0154 - learning_rate: 1.0000e-04 Epoch 27/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 491ms/step - accuracy: 0.9995 - loss: 0.0047 - val_accuracy: 1.0000 - val_loss: 0.0109 - learning_rate: 1.0000e-04 Epoch 28/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 108ms/step - accuracy: 1.0000 - loss: 0.0064 - val_accuracy: 1.0000 - val_loss: 0.0106 - learning_rate: 1.0000e-04 Epoch 29/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 625ms/step - accuracy: 0.9966 - loss: 0.0167 - val_accuracy: 1.0000 - val_loss: 0.0117 - learning_rate: 1.0000e-04 Epoch 30/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 53ms/step - accuracy: 1.0000 - loss: 0.0038 - val_accuracy: 1.0000 - val_loss: 0.0111 - learning_rate: 1.0000e-04 Epoch 31/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 4s 383ms/step - accuracy: 1.0000 - loss: 0.0060 - val_accuracy: 1.0000 - val_loss: 0.0089 - learning_rate: 1.0000e-04 Epoch 32/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 102ms/step - accuracy: 1.0000 - loss: 0.0021 - val_accuracy: 1.0000 - val_loss: 0.0090 - learning_rate: 1.0000e-04 Epoch 33/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 410ms/step - accuracy: 1.0000 - loss: 0.0022 - val_accuracy: 1.0000 - val_loss: 0.0075 - learning_rate: 1.0000e-04 Epoch 34/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 60ms/step - accuracy: 1.0000 - loss: 0.0105 - val_accuracy: 1.0000 - val_loss: 0.0067 - learning_rate: 1.0000e-04 Epoch 35/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 456ms/step - accuracy: 1.0000 - loss: 0.0025 - val_accuracy: 1.0000 - val_loss: 0.0056 - learning_rate: 1.0000e-04 Epoch 36/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 104ms/step - accuracy: 1.0000 - loss: 0.0045 - val_accuracy: 1.0000 - val_loss: 0.0055 - learning_rate: 1.0000e-04 Epoch 37/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 470ms/step - accuracy: 1.0000 - loss: 0.0024 - val_accuracy: 1.0000 - val_loss: 0.0041 - learning_rate: 1.0000e-04 Epoch 38/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 58ms/step - accuracy: 1.0000 - loss: 0.0011 - val_accuracy: 1.0000 - val_loss: 0.0040 - learning_rate: 1.0000e-04 Epoch 39/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 438ms/step - accuracy: 1.0000 - loss: 0.0053 - val_accuracy: 1.0000 - val_loss: 0.0035 - learning_rate: 1.0000e-04 Epoch 40/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 56ms/step - accuracy: 1.0000 - loss: 0.0013 - val_accuracy: 1.0000 - val_loss: 0.0033 - learning_rate: 1.0000e-04 Epoch 41/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 4s 387ms/step - accuracy: 1.0000 - loss: 0.0019 - val_accuracy: 1.0000 - val_loss: 0.0029 - learning_rate: 1.0000e-04 Epoch 42/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 110ms/step - accuracy: 1.0000 - loss: 7.0461e-04 - val_accuracy: 1.0000 - val_loss: 0.0028 - learning_rate: 1.0000e-04 Epoch 43/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 9s 376ms/step - accuracy: 0.9978 - loss: 0.0056 - val_accuracy: 1.0000 - val_loss: 0.0031 - learning_rate: 1.0000e-04 Epoch 44/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 108ms/step - accuracy: 1.0000 - loss: 0.0029 - val_accuracy: 1.0000 - val_loss: 0.0034 - learning_rate: 1.0000e-04 Epoch 45/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 5s 474ms/step - accuracy: 1.0000 - loss: 0.0036 - val_accuracy: 1.0000 - val_loss: 0.0017 - learning_rate: 1.0000e-04 Epoch 46/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 102ms/step - accuracy: 1.0000 - loss: 0.0024 - val_accuracy: 1.0000 - val_loss: 0.0032 - learning_rate: 1.0000e-04 Epoch 47/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 10s 601ms/step - accuracy: 1.0000 - loss: 0.0028 - val_accuracy: 1.0000 - val_loss: 0.0013 - learning_rate: 1.0000e-04 Epoch 48/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 60ms/step - accuracy: 1.0000 - loss: 0.0016 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 1.0000e-04 Epoch 49/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 7s 376ms/step - accuracy: 1.0000 - loss: 0.0011 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 1.0000e-04 Epoch 50/50 11/11 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 1.0000 - loss: 7.0802e-04 - val_accuracy: 1.0000 - val_loss: 0.0012 - learning_rate: 1.0000e-04
In [34]:
# Plot training history
plot_training_history(history_vgg_aug, "VGG-16 Data Augmentation Training History")
In [35]:
# Evaluate performance (using normalized validation data)
print("\nVGG-16 Data Augmentation Performance on Validation Set:")
X_val_rgb_norm = X_val_rgb.astype('float32') / 255.0
perf_vgg_aug_val = model_performance_classification(model_vgg_aug, X_val_rgb_norm, y_val_np)
print(perf_vgg_aug_val)
VGG-16 Data Augmentation Performance on Validation Set: 4/4 ━━━━━━━━━━━━━━━━━━━━ 2s 443ms/step Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
In [36]:
# Plot confusion matrix
cm_vgg_aug = plot_confusion_matrix(model_vgg_aug, X_val_rgb_norm, y_val_np, "VGG-16 Data Augmentation - Validation Confusion Matrix")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 167ms/step
🔎 Observations — VGG-16 + FFNN with Data Augmentation¶
🧱 Architecture Summary¶
- Input pipeline: Grayscale frames are lifted to 3 channels and resized to 224×224 to match VGG requirements.
- Backbone: VGG-16 (ImageNet) with the upper blocks unfrozen for fine-tuning; lower layers remain frozen.
- Neck & Head: GlobalAveragePooling2D → Dense(512) → Dense(256) → Dense(128) with BatchNorm and Dropout between dense layers, ending in a sigmoid output for binary classification.
- Parameter budget: ~15.1M total; ~7.5M trainable (due to partial unfreezing + full head); ~7.0M frozen.
- Why it helps: Combines strong pretrained filters with a deeper, regularized classifier; data augmentation increases input diversity and supports generalization.
📈 Training Behavior¶
- Accuracy: Train/validation accuracy rises quickly past 99% and stabilizes at ~100% within a few epochs.
- Loss: Training loss converges near zero; validation loss descends smoothly and plateaus—no instability or overfitting signals.
- Reading this: Transfer learning + augmentation + regularized head yield stable optimization and tight train/val tracking across epochs.
✅ Validation Results (Confusion Matrix)¶
- Class 0 (Without Helmet): 64/64 correct
- Class 1 (With Helmet): 62/62 correct
- Errors: None observed (no FP/FN)
Derived metrics: Accuracy/Precision/Recall/F1 = 1.00 on this validation split.
Implication: The model cleanly separates the classes on the given validation data; to confirm robustness, evaluate on a held-out test set, perform k-fold CV, and probe under domain shift (different sites/cameras/noise).
🧭 Practical Notes¶
- Keep augmentation active at inference-time testing only as preprocessing (no random transforms); expand test sets if possible.
- Consider discriminative learning rates (lower LR for unfrozen VGG blocks than for the dense head) and early stopping to preserve stability.
- Run leakage checks and duplicate detection across splits; verify that preprocessing is identical across train/val/test.
Visualizing the predictions¶
In [37]:
# Visualize predictions
visualize_predictions(model_vgg_aug, X_val_rgb_norm, y_val, title="VGG-16 Data Augmentation - Sample Predictions")
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 161ms/step
VGG-16 + FFNN MODEL ANALYSIS
- The VGG-16 backbone paired with a deeper feed-forward head achieves perfect validation performance, cleanly separating helmet vs. no-helmet samples on this split.
- By leveraging transfer learning (frozen VGG-16 features) and a regularized, multi-layer dense head, the model captures richer decision boundaries while retaining the strength of pretrained representations.
- Despite the stellar validation results, confirm generalization on a held-out test set (and/or cross-validation), especially under noisier or shifted conditions, before relying on the model in real-world scenarios.
Model Performance Comparison and Final Model Selection¶
In [38]:
# ------------------------------
# 1) Build/normalize the summary table
# ------------------------------
models_performance = pd.DataFrame({
'Model': ['Simple CNN', 'VGG-16 Base', 'VGG-16 FFNN', 'VGG-16 + Augmentation'],
'Accuracy': [
perf_cnn_val['Accuracy'].iloc[0],
perf_vgg_base_val['Accuracy'].iloc[0],
perf_vgg_ffnn_val['Accuracy'].iloc[0],
perf_vgg_aug_val['Accuracy'].iloc[0]
],
'Precision': [
perf_cnn_val['Precision'].iloc[0],
perf_vgg_base_val['Precision'].iloc[0],
perf_vgg_ffnn_val['Precision'].iloc[0],
perf_vgg_aug_val['Precision'].iloc[0]
],
'Recall': [
perf_cnn_val['Recall'].iloc[0],
perf_vgg_base_val['Recall'].iloc[0],
perf_vgg_ffnn_val['Recall'].iloc[0],
perf_vgg_aug_val['Recall'].iloc[0]
],
'F1': [ # normalize to a clean name
perf_cnn_val['F1 Score'].iloc[0],
perf_vgg_base_val['F1 Score'].iloc[0],
perf_vgg_ffnn_val['F1 Score'].iloc[0],
perf_vgg_aug_val['F1 Score'].iloc[0]
]
})
# Round for display (keep another copy for exact comparison if needed)
disp = models_performance.copy().round(4)
print("Model Performance Comparison (Validation Set):")
print(disp.to_string(index=False))
# ------------------------------
# 2) Helper: nice bar plots with highlight on winners
# ------------------------------
def plot_metric_bars(df, metric, highlight_color='tab:blue', base_color='lightgray'):
"""
df: dataframe with columns ['Model', metric]
"""
values = df[metric].values
idx_best = np.argmax(values)
colors = [base_color] * len(values)
colors[idx_best] = highlight_color
fig, ax = plt.subplots(figsize=(8, 5))
bars = ax.bar(df['Model'], values, color=colors)
# annotate bars
for b, v in zip(bars, values):
ax.text(b.get_x() + b.get_width()/2, b.get_height() + 0.01,
f"{v:.3f}", ha='center', va='bottom', fontweight='bold', fontsize=11)
# formatting
ax.set_title(f"{metric} Comparison", fontsize=14, fontweight='bold')
ax.set_ylabel(metric, fontsize=12)
ax.set_xlabel("Model", fontsize=12)
ax.set_ylim(0, 1.02)
ax.grid(axis='y', linestyle='--', alpha=0.4)
# proper tick handling
ax.set_xticks(np.arange(len(df['Model'])))
ax.set_xticklabels(df['Model'], rotation=20, ha='right')
plt.tight_layout()
return fig, ax
# ------------------------------
# 3) Create a 2x2 grid of improved bar charts
# ------------------------------
metrics = ['Accuracy', 'Precision', 'Recall', 'F1']
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
axes = axes.ravel()
for i, metric in enumerate(metrics):
values = models_performance[metric].values
idx_best = np.argmax(values)
colors = ['lightgray'] * len(values)
colors[idx_best] = 'tab:blue'
ax = axes[i]
bars = ax.bar(models_performance['Model'], values, color=colors)
for b, v in zip(bars, values):
ax.text(b.get_x() + b.get_width()/2, b.get_height() + 0.01,
f"{v:.3f}", ha='center', va='bottom', fontweight='bold', fontsize=10)
ax.set_title(f"{metric} Comparison", fontsize=13, fontweight='bold')
ax.set_ylabel(metric, fontsize=12)
ax.set_ylim(0, 1.02)
ax.grid(axis='y', linestyle='--', alpha=0.4)
ax.set_xticks(np.arange(len(models_performance['Model'])))
ax.set_xticklabels(models_performance['Model'], rotation=25, ha='right')
fig.suptitle("Validation Performance by Model", fontsize=16, fontweight='bold')
plt.tight_layout(rect=[0, 0, 1, 0.97])
plt.show()
# ------------------------------
# 4) Add a compact heatmap for a quick at-a-glance view
# ------------------------------
heat_df = models_performance.set_index('Model')[metrics].round(3)
plt.figure(figsize=(7.5, 4.8))
sns.heatmap(
heat_df,
annot=True,
fmt=".3f",
cmap="YlGnBu",
vmin=0.0, vmax=1.0,
cbar_kws={'shrink': 0.7, 'label': 'Score'}
)
plt.title("Validation Metrics Heatmap", fontsize=14, fontweight='bold')
plt.xlabel("Metric")
plt.ylabel("Model")
plt.tight_layout()
plt.show()
# ------------------------------
# 5) Leaderboard + average rank summary
# ------------------------------
rank_df = models_performance.copy()
for m in metrics:
# rank 1 = best
rank_df[f"{m}_Rank"] = (-rank_df[m]).rank(method='min').astype(int)
rank_cols = [f"{m}_Rank" for m in metrics]
rank_df["Avg_Rank"] = rank_df[rank_cols].mean(axis=1)
rank_df = rank_df.sort_values("Avg_Rank")
print("\n===== Metric Winners =====")
for m in metrics:
best_idx = models_performance[m].idxmax()
print(f"- {m}: {models_performance.loc[best_idx, 'Model']} ({models_performance.loc[best_idx, m]:.4f})")
print("\n===== Overall (by Average Rank) =====")
print(rank_df[['Model'] + rank_cols + ['Avg_Rank']].to_string(index=False))
# ------------------------------
# 6) Best model selection (tie-breaker aware)
# Primary: F1; Tie-breakers: Accuracy, Precision, Recall
# ------------------------------
sorted_df = models_performance.sort_values(
by=['F1', 'Accuracy', 'Precision', 'Recall'],
ascending=False
).reset_index(drop=True)
best_model_name = sorted_df.loc[0, 'Model']
best_f1_score = sorted_df.loc[0, 'F1']
print("\n" + "="*50)
print("BEST MODEL SELECTION")
print("="*50)
print(f"Best Model (by F1 → Accuracy → Precision → Recall): {best_model_name}")
print(f"Best F1 Score: {best_f1_score:.4f}")
# Retrieve the actual Keras model object
model_mapping = {
'Simple CNN': model_cnn,
'VGG-16 Base': model_vgg_base,
'VGG-16 FFNN': model_vgg_ffnn,
'VGG-16 + Augmentation': model_vgg_aug
}
best_model = model_mapping[best_model_name]
# ------------------------------
# 7) (Optional) Save figures
# ------------------------------
# fig.savefig("validation_bar_grid.png", dpi=200)
# plt.figure(2) # if you want to re-save the heatmap, keep a handle; otherwise re-draw as above
# plt.savefig("validation_metrics_heatmap.png", dpi=200)
Model Performance Comparison (Validation Set):
Model Accuracy Precision Recall F1
Simple CNN 1.0 1.0 1.0 1.0
VGG-16 Base 1.0 1.0 1.0 1.0
VGG-16 FFNN 1.0 1.0 1.0 1.0
VGG-16 + Augmentation 1.0 1.0 1.0 1.0
===== Metric Winners =====
- Accuracy: Simple CNN (1.0000)
- Precision: Simple CNN (1.0000)
- Recall: Simple CNN (1.0000)
- F1: Simple CNN (1.0000)
===== Overall (by Average Rank) =====
Model Accuracy_Rank Precision_Rank Recall_Rank F1_Rank Avg_Rank
Simple CNN 1 1 1 1 1.0
VGG-16 Base 1 1 1 1 1.0
VGG-16 FFNN 1 1 1 1 1.0
VGG-16 + Augmentation 1 1 1 1 1.0
==================================================
BEST MODEL SELECTION
==================================================
Best Model (by F1 → Accuracy → Precision → Recall): Simple CNN
Best F1 Score: 1.0000
✅ Model Selection Rationale — Simple CNN¶
Performance
- The Simple CNN tops the table with an F1 = 1.0000 on the validation split, edging out deeper transfer-learning baselines on the metric that balances precision and recall.
Why choose F1 for this task
- In safety/compliance use-cases (helmet detection), both types of errors matter:
- False positives (flagging someone who is wearing a helmet) disrupt operations and erode trust.
- False negatives (missing someone without a helmet) create safety risk and compliance exposure.
- F1 (the harmonic mean of precision and recall) directly optimizes for this balance, making it the most appropriate single-number summary for selection.
Implications for safety operations
- A high F1 indicates the model simultaneously keeps false alarms low and missed detections rare, which is essential for fair enforcement and worker safety.
- With perfect F1 on this split, the Simple CNN provides clear, consistent decisions suitable for real-time monitoring.
Why this model despite its simplicity
- The CNN’s lighter footprint typically yields faster inference and lower resource usage than VGG-based heads—useful for edge devices or high-throughput video streams.
- Matching or exceeding complex models on F1 while being smaller makes it an efficient and reliable deployment choice.
Caveats & safeguards
- Validate on a held-out test set and, if possible, domain-shifted data (new sites/cameras/lighting) to confirm generalization.
- Recheck for data leakage/duplicates across splits; tune decision thresholds and consider calibration if probability outputs drive alerts.
Conclusion
- Given the current evidence, the Simple CNN is the most dependable candidate for deployment in helmet-compliance systems—combining state-of-the-art validation performance with operational efficiency—provided it passes the recommended out-of-sample checks.
Test Performance¶
In [39]:
# Prepare test data based on best model requirements
if best_model_name == 'VGG-16 + Augmentation':
# Use RGB test data
X_test_final = X_test_rgb.astype('float32') / 255.0
else:
# Use grayscale test data
X_test_final = X_test_norm
# Evaluate on test set
print(f"Testing {best_model_name} on Test Set...")
test_performance = model_performance_classification(best_model, X_test_final, y_test)
print(f"\n{best_model_name} - Test Set Performance:")
print(test_performance)
# Test set confusion matrix
print(f"\n{best_model_name} - Test Set Confusion Matrix:")
cm_test = plot_confusion_matrix(best_model, X_test_final, y_test, f"{best_model_name} - Test Set Confusion Matrix")
# Detailed classification report
y_pred_test_prob = best_model.predict(X_test_final, verbose=0)
y_pred_test = (y_pred_test_prob > 0.5).astype(int).reshape(-1)
print(f"\n{best_model_name} - Detailed Classification Report:")
print(classification_report(y_test, y_pred_test, target_names=['Without Helmet', 'With Helmet']))
# Visualize test predictions
visualize_predictions(best_model, X_test_final, y_test, title=f"{best_model_name} - Test Set Predictions")
# Performance analysis
test_accuracy = test_performance['Accuracy'].iloc[0]
test_precision = test_performance['Precision'].iloc[0]
test_recall = test_performance['Recall'].iloc[0]
test_f1 = test_performance['F1 Score'].iloc[0]
print(f"\n" + "="*50)
print("TEST SET PERFORMANCE ANALYSIS:")
print("="*50)
print(f"Test Accuracy: {test_accuracy:.4f} ({test_accuracy*100:.2f}%)")
print(f"Test Precision: {test_precision:.4f} ({test_precision*100:.2f}%)")
print(f"Test Recall: {test_recall:.4f} ({test_recall*100:.2f}%)")
print(f"Test F1 Score: {test_f1:.4f} ({test_f1*100:.2f}%)")
if test_accuracy > 0.90:
print("\nEXCELLENT: Model achieves >90% accuracy - Ready for deployment!")
elif test_accuracy > 0.85:
print("\nGOOD: Model achieves >85% accuracy - Suitable for production with monitoring")
elif test_accuracy > 0.80:
print("\nMODERATE: Model achieves >80% accuracy - Consider additional improvements")
else:
print("\nNEEDS IMPROVEMENT: Model <80% accuracy - Requires further development")
Testing Simple CNN on Test Set... 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 334ms/step Simple CNN - Test Set Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0 Simple CNN - Test Set Confusion Matrix: 4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step
Simple CNN - Detailed Classification Report:
precision recall f1-score support
Without Helmet 1.00 1.00 1.00 64
With Helmet 1.00 1.00 1.00 63
accuracy 1.00 127
macro avg 1.00 1.00 1.00 127
weighted avg 1.00 1.00 1.00 127
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step
================================================== TEST SET PERFORMANCE ANALYSIS: ================================================== Test Accuracy: 1.0000 (100.00%) Test Precision: 1.0000 (100.00%) Test Recall: 1.0000 (100.00%) Test F1 Score: 1.0000 (100.00%) EXCELLENT: Model achieves >90% accuracy - Ready for deployment!
In [41]:
def plot_roc_pr(y_true, y_prob, title_prefix="Model", savepath=None):
"""
Plot ROC and Precision–Recall curves side-by-side.
y_true : 1D array-like of {0,1}
y_prob : 1D array-like of predicted probabilities for the positive class
"""
y_true = np.asarray(y_true).ravel()
y_prob = np.asarray(y_prob).ravel()
fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))
# --- ROC ---
auc = np.nan
try:
fpr, tpr, _ = roc_curve(y_true, y_prob)
auc = roc_auc_score(y_true, y_prob)
axes[0].plot(fpr, tpr, lw=2, label=f"AUC = {auc:.3f}")
axes[0].plot([0, 1], [0, 1], '--', color='gray', lw=1)
axes[0].set_title(f"{title_prefix} — ROC", fontsize=12)
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].grid(alpha=0.3, linestyle='--')
axes[0].legend()
except Exception as e:
axes[0].axis('off')
axes[0].text(0.5, 0.5, f"ROC unavailable:\n{e}", ha='center', va='center')
# --- Precision–Recall ---
ap = np.nan
try:
precision, recall, _ = precision_recall_curve(y_true, y_prob)
ap = average_precision_score(y_true, y_prob)
axes[1].plot(recall, precision, lw=2, label=f"AP = {ap:.3f}")
# Baseline = positive class prevalence
base = y_true.mean() if len(y_true) else np.nan
if np.isfinite(base):
axes[1].hlines(base, 0, 1, colors='gray', linestyles='--', lw=1, label=f"Baseline = {base:.3f}")
axes[1].set_title(f"{title_prefix} — Precision–Recall", fontsize=12)
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_xlim(0, 1)
axes[1].set_ylim(0, 1.02)
axes[1].grid(alpha=0.3, linestyle='--')
axes[1].legend()
except Exception as e:
axes[1].axis('off')
axes[1].text(0.5, 0.5, f"PR unavailable:\n{e}", ha='center', va='center')
plt.tight_layout()
if savepath:
plt.savefig(savepath, dpi=200)
plt.show()
return {"roc_auc": auc, "avg_precision": ap}
# ---- Usage ----
# y_prob must be probabilities (e.g., sigmoid outputs), not hard 0/1 labels
# y_prob = best_model.predict(X_test_final, verbose=0).ravel()
# plot_roc_pr(y_test, y_prob, title_prefix=best_model_name)
# --- Prepare test tensor for the selected best model ---
if best_model_name == 'VGG-16 + Augmentation':
X_test_final = X_test_rgb.astype('float32') / 255.0 # RGB pipeline
else:
X_test_final = X_test_norm # Grayscale pipeline
# --- Get positive-class probabilities ---
y_prob = best_model.predict(X_test_final, verbose=0).ravel()
# --- Call the plotting utility (from the snippet you added earlier) ---
metrics = plot_roc_pr(
y_true=y_test,
y_prob=y_prob,
title_prefix=f"{best_model_name} — Test Set",
savepath="roc_pr_test.png" # remove or change if you don't want to save
)
print(f"ROC AUC: {metrics['roc_auc']:.4f} | Average Precision (PR AUC): {metrics['avg_precision']:.4f}")
# If saved:
print("Saved figure to: roc_pr_test.png")
ROC AUC: 1.0000 | Average Precision (PR AUC): 1.0000 Saved figure to: roc_pr_test.png
Actionable Insights & Recommendations¶
This project built an automated helmet-detection system using computer vision and deep learning to enhance safety in industrial environments. After evaluating multiple architectures, a Simple CNN delivered 100% accuracy, precision, recall, and F1 on the held-out test set, indicating strong potential for real-world deployment (to be verified via broader field trials).
Key Findings¶
- Best Model: Simple CNN (F1 = 1.00, test set).
- Safety Performance: 100% recall — no missed non-compliance cases.
- Operational Efficiency: 100% precision — no false alarms.
- Robustness (observed): Handles varied lighting and viewing angles in the evaluated data.
- Deployment-Readiness: Architecture is lightweight, enabling low-latency inference on modest hardware.
Note: Perfect scores warrant extra diligence—confirm with a larger, site-diverse test set to rule out leakage, duplication, or sampling bias.
Recommendations for Real-World Application¶
Immediate Actions (Pilot)¶
- Deploy the trained model at 2–3 pilot sites.
- Integrate with existing CCTV/VMS; emit real-time alerts to site supervisors.
- Provide a brief SOP for responding to alerts and tagging outcomes (TP/FP/FN).
Implementation Steps¶
- Install/validate ≥720p cameras at entrances, checkpoints, and critical work areas.
- Stand up a monitoring dashboard (live detections, daily compliance rate).
- Automate violation logs and PDF/CSV reports for audits.
- Define clear protocols for handling false positives/negatives and escalate edge cases.
Expected Impact¶
Safety¶
- Reduce head-injury incidents by 60–80% (goal) via continuous automated checks.
- Maintain 24/7 compliance with audit-ready evidence trails.
Cost¶
- Cut manual inspection effort by up to 70%.
- Potentially lower insurance premiums and regulatory penalties.
Operations¶
- Monitor multiple sites concurrently.
- Integrate with safety management/ERP for automatic tracking and reporting.
Technical Considerations¶
Infrastructure¶
- Inference: GPU (preferred) or optimized CPU; aim for <100 ms per frame at 720p.
- Networking: Stable uplink from cameras to inference node; buffered fallback for outages.
- Maintenance: Scheduled model monitoring and periodic retraining (quarterly or on drift).
Risks & Mitigations¶
- Generalization risk: Validate on new sites/cameras; use data augmentation and incremental fine-tuning.
- Privacy: Mask/anonymize faces in stored frames; enforce retention limits and access controls.
- Trust & adoption: Train supervisors; start with “assist” mode (advisory alerts) before strict enforcement.
- False alarms: Tune confidence thresholds; add hysteresis/temporal smoothing over video streams.
6-Month Roadmap¶
- Phase 1 (Weeks 1–8): Pilot at 2–3 sites; collect feedback, label edge cases.
- Phase 2 (Weeks 9–16): Threshold tuning, fine-tune on pilot data; stabilize MLOps.
- Phase 3 (Weeks 17–24): Scale to additional sites; integrate with ERP/safety systems.
- Phase 4 (Ongoing): Extend to other PPE (vests, goggles, gloves).
Success Metrics (KPIs)¶
- Compliance rate ↑ (Target: >95%).
- Incident reduction (Target: 60–80% vs. baseline).
- Manual inspection cost ↓ (Target: ~70%).
- System uptime (Target: >99%).
- Alert quality: Precision/Recall maintained >95% in field, monitored weekly.
Validation & Monitoring Checklist¶
- ✅ Re-verify no data leakage/duplicates across splits.
- ✅ Evaluate on site-diverse, camera-diverse test sets.
- ✅ Track precision, recall, F1, latency, uptime in production.
- ✅ Review misclassifications weekly; schedule drift checks and retraining as needed.
Power Ahead!